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Introduction 

This is part three in this series of papers. Throughout this paper, A; is a field of 
characteristic zero. In parts one and two [25], [23], we consider cases (2), (5), (6), (7) 
in the Sato-Kimura classification [19]. In this part, we consider the prehomogeneous 
vector space G = GL(5) x GL(3), V = A^k^ ® k^. This is rather an isolated case 
and that's why we consider this single case separately here. However, as far as the 
relation between the theory of prehomogeneous vector spaces and ergodic theory 
is concerned, this may be the most interesting case besides the situation of the 
Oppenheim conjecture G = GL(n), V = Sym (A;")*, because it produces a family 
of (irrational) cubic forms in five variables whose values at integer points are dense 
in M. This may be the family of cubic forms in the lowest number of variables we 
can achieve by the theory of prehomogeneous vector spaces. We consider most of 
the remaining irreducible split cases in part four. 

Oppenheim conjectured in [12] that if Q{x) is a real non-degenerate indefinite 
quadratic form in n > 5 variables such that the ratio of at least one pair of coef- 
ficients is irrational, for any e > 0, there exists a; G Z" such that < |Q(a;)| < e. 
Due to the result of Lewis [9], this is equivalent to say the set {Q{x) \ x G Z"} is 
dense in M. There were many partial results including the one by Davenport with 
the collaboration with others ([4], [5], [6], [2], [18]) for n > 21. It was proved in the 
optimum form by Margulis (see [10]) for n > 3 using ergodic theory. 

We posed the question of generalizing the Oppenheim conjecture from the view- 
point of prehomogeneous vector spaces in [25]. For more detailed comments, the 
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reader should see the introduction of [25]. Here, we briefly state what we are going 
to prove. 

Let Hi = SL{5),H2 = SL{3),H = Hi x H2. It is known that (G, F) is a pre- 
homogeneous vector space (see [22] , [26] , [25] for the deflnition of prehomogeneous 
vector spaces). A non-constant polynomial A{x) on V is called a relative invariant 
polynomial if there exists a character x such that A{gx) = x{9)A.{x). Such A{x) 
exists for our case and is essentially unique. So we deflne V^^ = {x & V \ A{x) j^ 0} . 
For X e V^^, let H^r_^_ be the connected component of 1 in classical topology of the 
stabilizer H^r- We will prove that if a; e V^^ is "sufficiently irrational", H^^_^_Hz 
is dense in Hr. 

What Margulis did was to prove the above statement for the case H = SL{3),V = 
Sym (M^)*. Our method is based on the following theorem due to Ratner. 

Theorem (0.1) (Ratner) Let G be a connected Lie group and U a connected 
subgroup of G generated by unipotent elements of G. Then given any lattice T <Z G 
and X e G/V , there exists a connected closed subgroup U C F C G such that 
UxT = FxT. Moreover, F/F T has a finite invariant measure. 

Note that in the above theorem, the definition of a lattice contains the condition 
that G/T has a finite volume. The first statement was called Raghunathan's topo- 
logical conjecture, and the second statement was proved by Ratner in conjunction 
with Raghunathan's topological conjecture. Raghunathan's topological conjecture 
was published by Dani [3] for one dimensional unipotent groups and was general- 
ized to groups generated by unipotent elements by Margulis [10] (before Ratner's 
proof). The proof for the general case was given by Ratner in a series of papers 
[13], [14], [15], [16]. For these, there is an excellent survey article by Ratner [17]. 

Note that in the above theorem, if G is an algebraic group over Q and F is an 
arithmetic lattice, the group F becomes an algebraic group defined over Q. For 
this, the reader should see Proposition (3.2) [20, pp. 321-322]. It is also proved in 
Proposition (3.2) [20, pp. 321-322] that the radical of F is a unipotent subgroup. 
In [20], only one lattice is considered, but one can deduce the above statement 
for any lattice commensurable with the lattice in [20] by a simple argument using 
Ratner's theorem. 

We describe our result explicitly here. For any non-zero point a; in a vector space, 
we denote the point in the corresponding projective space determined by x by [x]. 
Also if 5 is a subspace of a vector space, we denote the point in the Grassmann 
variety determined by S by [S] also. Let Vi be a five dimensional vector space 
defined over Q. We fix a rational basis {mo, • • • , m^} for Vi. Let S be the subspace 
of A^Fi spanned by the following three elements 

nil A m^ — 3m2 A m^, 
mo A m^ — 2mi A m^, 
TJiQ A 7723 — 3mi A m2. 

This space is of dimension three and therefore, determines an element [S] in the 
Grassmann Gr(3, Vi) = Gr(3, 10) of three dimensional subspaces in Vi. Because of 
the rational structure of Vi, Gr(3, Vi) has a rational structure also. 



Let 



Q{a) = 00^4 — 701^3 + 77Q25 
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F{a) = 72ooa2a4 + 9oia203 — 202 — 21aQa\ — 21a\a4^ 



for o = '^i^Qdimi. If we identify Vi with the space of binary quartic forms by 
a -^ movf + • • • + m^V2, Q,F correspond to quadratic and cubic SL(2)-invariant 
polynomials. Also [S] E Gr(3, 10) is the unique fixed point of SL(2). 

li g e GL(yi)R ^ GL(5)r, it naturally acts on Gr(3, Fi)r, P(Sym2yi*)R, and 
P(SymVi*)R. Note that {gQ){a) = Qig'^a), {gF){a) = F{g-^a). Then the 
following theorem is the main result of this paper. 

Theorem (0.2) Suppose g[S] ^ Gr(3, Fi)q and g[Q] ^ P(SymVi*)Q. Then the set 
of values of the cubic polynomial F {g~^ a) at primitive integer points inll" is dense 
in"^. 

In §1, we consider various identifications concerning tensor products of vector 
spaces. If a; e V^ -, by Ratner's theorem, there exists a closed connected subgroup 
iy^R_,_ C F C Hs. such that H^^_^Hz = FHi. In §2, we construct equivariant maps 
from V^^ to various i^-varieties. In §3, we prove that these equivariant maps are 
well defined and are non-trivial. These equivariant maps correspond to families 
of such F's with the property that if Xp is the corresponding i^-variety, F has 
a unique fixed point in Xp- Part of our consideration resembles the argument in 
[21]. In §4, we describe the orbit space to determine when H^^j^ is generated by 
unipotent elements. In §5, we classify all F's as above. In §6, we prove Theorem 
(0.2). 

§1 Preliminaries 

We are going to do a lot of computations in §3 regarding symmetric tensor 
products of vector spaces. We fix various normalizations for that purpose in this 
section. 

Let 1^ be a vector space over k with a basis {ei, • • • , e„}. Let W* be the dual 
space with the dual basis {/i, • • • , /„}. For ai, • • • , a^ G W ^ we define 

[ai,--- ,ad]d = ^ XI "'^(1) ® ■■■® «T(d), 

where &d is the group of permutations of {1, • • • ,d}. We identify Sym W with 
the subspace of 1^®'^ spanned by elements of the form [ai, • • • , a^j^. Similarly, we 
identify Sym W* with a subspace of (W*)®'^. For ai, • • • , a^^, fldi+i, • • • , (idi+d2 ^ 
W , we define 

[oi, • • • ,adi]di[adi+i, • • • ,adi+d2\d2 = [^i, • • • ,ad^+d2\d'i_+d2- 

By this product, ©Sym*!^ becomes an associative algebra. Since [ai, • • • , ad\d = 
ai • • • Qd, we use this usual notation of product from now on. 

Since (W*)®'^ and W®'^ are dual spaces of each other, there is a natural pairing 
between Sym'^W and Sym'^W* . If a = ai • • • a^ e Sym'^W, b = bi---bde Sym'^W* , 



we normalize this pairing by 

(o, fe)d= (fe, o)d= — ^ bi{aa(i))---bdiaa(d))- 






Then if ii + • • • + i„ = c?, 



(er •••<", /r---/^)d 



t„ fti j'i.„\ 1l- ' ' 'In- 
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Therefore, Sym W* can be identified with the dual space of Sym W . 
The map 

W 3 a -f id{a) = a(g)a(g)---(g)aG Sym'^W 

is a polynomial map. So if / G Sym W* , f{a) = f{id{a)) is a polynomial map from 
W to k and is homogeneous of degree d. We can identify Sym W* with the space 
of degree k forms on W by this correspondence. If o = Yl7=i '^i^i^ 

^1- ■ ■ ■ ^n- 

where the sum is over all (ii, • • • , in) such that ii + • • - + 271 = c^- So if / = /|^ • • • /^" , 
/(a) = a^^ • • • a^". Therefore, / corresponds to the monomial a^^ • ■ • a^". 

If G C GIj{W) is a subgroup, G acts on W* by {gf){v) = f{g~^v) for g^ e 
G,fE W* . Whenever we consider the contragredient representation, we consider 
this action. 

§2 Definitions of equivariant maps 

Let G, V, H be as in the introduction. We construct iif-equivariant maps from 
V or V^^ to various iif-varieties in this section. 

Let 1^ = A;^ be the space of two dimensional column vectors. Let {61,62} be 
the standard basis of W . Consider the usual action of GL(2) on W . This induces 
an action of GL(2) on Sym W and Sym W* for any d. We define new actions of 
GL(2) on Sym^VF, Sym W hy g ■ x = {deig)~^gx, {detg)~^gx where ^r • a; is the 
new action and gx is the usual action. Note that scalar matrices act trivially and 
therefore, this defines an action of PGL(2) on Sym W, Sym W . 

Let Vi = Sym^VT, V2 = Sym^VT, V = A^Vi ® V2 and d = GL(yi) ^ 
GL(5), G2 = GL(y2) = GL(3), G = Gi x G2. Then G acts on V in the usual 
manner and the above action of PGL(2) defines a homomorphism PGL(2) -^ G. By 
Schur's lemma, this is an imbedding. In fact, Ker(PGL(2) —)■ Gi) = Ker(PGL(2) -^ 
G2) = {!}. So we regard PGL(2) as a subgroup of G. Let f = Ker(G -^ GL(F)). 
By Schur's lemma again, 

T = {{th, t-^h) I t e GL(l)} ^ GL(1). 

If (^75,^-2/3) G PGL(2), it acts trivially on Fi,F2- So t = 1. Therefore, PGL(2) n 
f = {l}. 



Let Iq = e1,li = 6162,^2 = e| and mo = ef , • • • , m4 = e|. Then {loJijh}, 
{mo, • • • , m,^} are bases of V2, Vi respectively. 

We define a linear map </>i : F ^ A'^Fi ® Sym^y2 = V^^* Sym^F2 by 



N N 

(2.1) V3^Pi®qi^-^PiApj(S}qi 

i=l i,j = l 



for pi,--- ,PAr 6 A^Vi, qi,--- ,qN ^ V2. Regarding Sym^V2 as a subspace of 
V2 ® V2, we denote the element of Vi ® F2 ® V2 which corresponds to 0i(a;) by 
(f)i{x). Regarding V^ as the contragredient representation of Vi, Vi Sym^Vi is a 
representation of G. 

The following lemma can be proved as in [19, p. 80], and the proof is left to the 
reader. 

Lemma (2.2) Forg = {gi,g2) G C, (j)i{gx) = detgig<pi{x), <pi{gx) = det gig(f)-^{x). 
li X = ^o<j<i<4 "^« ^ "^i ® ^^i "^^^^ ^ij ^ ^2, 

(l>i{x) = - ^ ^ miAmjAmkAmi^XijXki- 

0<i<j<4 0<fc</<4 

Let mg, • • • , ml e A^Fi be elements such that m,i A m,j = Sijm,o A • • • A m,^ {5ij 
is Kronecker's delta). Explicitly, 

mo = mi A 1712 A m^ A m^, mi = —mo A m,2 A m^ A m,^, etc. 

We identify A^Fi with the dual space of Vi by the pairing 

Vi X a'^Fi 3 {a,b) ^aAb 

and choosing mo A • • • A m^ as the basis element of A^Fi. Then {m,^, • • • , m|} can 
be regarded as the dual basis of {m,o, • • • , m,^}. 
Let 

a;i2a;34 - a;i3a;24 + a;i4a;23, 
-(3^023:34 - a;o3a;24 + 2:043:23), 
3:^013^34 - 2:033^14 + 2:043^13, 

— (a:oia^24 — Xo2Xl4: + 2:043^12) 

a:oia:23 - a;o2a:i3 + a;o3a:i2- 



(2.3) 


Pfaffo(x) 




Pfaffi(a;) 




Pfaff2(2;) 




Pfaff3(a;) 




Pfaff4(a;) 


Then 




(2.4) 


h 



(f)i{x) = ^m* ® Pfaffi(a;). 



The quadratic polynomials Pfaffo(a;), • • • , Pfaff4(a;) are the Pfaffians of 4x4 main 
minors of x if we regard x as an alternating 5x5 matrix with entries in Vi . This 



idea was used in [24] for the case G = GL(5) x GL(4), V = A^k^<^k'^ to parametrize 
quintic extensions of a given ground field. 

Next we consider a linear map 02 : Vi (g) Sym^y2 — > A^Sym^V2 — Sym^F2* 
defined by 

4 

(2.5) Vi (g) Sym^F2 3 "^m* pi ^ po A ■ ■ ■ A p4 E A'^Sym^y2 = Sym^T^s*- 



Definition (2.6) $i = ^</>2 o </>i. 

$1 is a map from V to Sym^V2*. The following lemma can also be proved as in 
[19, p. 80], and the proof is left to the reader. 

Lemma (2.7) ^i{gx) = (det5'i)^(det5f2)fl'2$i(a:^)- 

This map $i was also considered in [11] (using 0i also) for a different purpose. 
We will show in §3 that the discriminant of ^i{x) is not identically zero and V^^ 
consists ofay's such that ^i{x) is non-degenerate. 

For X E V, let ^i{x){a, 0) be the symmetric bilinear form on V2 associated with 
^i{x). In other words, 

^iix){a,f3) = ^{<^iix){a + f3)-^iix){a)-<^i{x){f3)) 

for a,f3 e V2. 

We define a linear map j^ : V{ ® F2 ® V'2 ^ Hom(F2, ^1* ® ^^2) by 

(2.8) i^ (a ® a ® /^) (7) = $1 (x) {a, 7)0 ® /3 

for aeV^". a. p. 'ye V2. If f E Hom(y2. ^^1*0^^2). we define gf E Hom(F2, ^i* 0^2) 
by 

(f//)(a) =9f{92^(^) 

where we are considering the action of gi on the element f{g2 o;). 

Lemma (2.9) jgx{g{a a j3)) = {det gi)^{det g2)gjx{a a P) for all a E 

V{,a,pEV2. 

Proof. Let 7 e V2- Then by Lemma (2.7), 

jgx{gia®a0 f3)){'y) = <^i{gx){g2a,^)gia® g2l3 

= {dei gi)'^{dei g2){g2^i{x)){g2a,l)gia ® g2P 
= (det ^i)^(det 5f2)$i {x){a, ^2'S)5'i« ® g2p 
= (det c/i)^(det 52)5(^1 {x){a, g2^-f)a ® /5) 
= {det gi)'^{det g2)g{jxia (» a (» fi){g^^^)) 
= {detgif{detg2){g{jx{a.®a®0))){-f). 

This proves the lemma. D 



Definition (2.10) M^) = j^iM^))- 

Apparently, ^3 is a map from V to Hom(V2, Vi ® V2). 
Lemma (2.11) </>3(5'x) = {detgi)^{detg2)94>3{x)- 

Proof. 

higx) = jgx((j)i{gx)) 

= <ietgijg^{g'(j)-^{x)) 

= {det gif {det g2)gjx((piix)) 

= {detgi)^{detg2)g(t)3{x). 

D 

By the basis {Iq, h, h} for V2, we can regard ^3(3;) as a 3 x 3 matrix witli entries 
in Vi . Then the action of ^r = (fi'1,5'2) G GL(Vi) x GL(3) is obtained by considering 
92'Pz{x)g2 and then applying gi entry- wise. Therefore, 

(2.12) $2(rK) = i^ihixf) e SymVi*, F, = det cf>s{x) e Sym^F^* 

define maps x -^ ^2{x), F^ from V to Sjm^V{ ,Sjm^V{ . 
The following lemma is an easy corollary of Lemma (2.11). 

Lemma (2.13) (1) ^2{gx) = {detgiY^idet g2)'^gi^2{x). 
(2) Fg^ia) = (det^i)i'5(det^2)'i^.(^r'a) for all a G Vi. 

For later purposes, we describe how to compute ^2{x),Fx. We have already 
described how to compute (f)i{x),^i{x) in (2.4), (2.5). Let {po,Pi,P2} be the dual 
basis of {/o, h, h}- Suppose 

2 2 

i,j=0 t,s=0 

with Gij G F]*, 6ts G ^ for all i,j,t,s and Ojj = Ojj, fet^ = bgt- We denote the 
matrices (uij), (bts) also by </>i(x), $i(x). Let 7 = ^^^qTs^s- Then 

2 
= X j^{aij (g) /i (g) lj){jsh) 

i,j,s=0 
2 

= X $i(a;)(/i(g)/s)ay7,. (g)/j 

i,j,s=0 
2 

= J^ aijbis^s ® /j 

i,j,s=0 
2 

= X! Cijihsls®lj- 
i,j,s=0 



Note that ^j^q '^ji^is is the {j, s)-entry of the matrix product (f)i{x)^i{x). So if 
we regard (psix) as a 3 x 3 matrix with entries in V{, we get the following relation 

(2.14) Mx) =^^{x)'^i{x). 

We can express any x E V as x = xo^lo+xi<^li+X2^l2 where a;o,a;i,a;2 ^ A^Vi. 
Let 5a; C A^Fi = k^^ be the subspace spanned by a;o,a;i,a;2- We will show in §3 
that dim^a; = 3 if a; G V^^. So if a; G V^^, we define 

(2.15) $3(2;) eGr(3, 10) 

to be the element of the Grassmann determined by Sx- This defines a map x -^ 
$3(3;) from V^^ to Gr(3, 10). The group Gi still acts on Gr(3, 10) and the following 
lemma is obvious. 

Lemma (2.16) ^sigx) = gi^six). 

§3 Equi variant maps at w 

In this section, we prove that the equivariant maps we constructed in §2 are well 
defined and are non-trivial by evaluating them at a point w, which we will define 
in (3.10). 

Let {^0, ^1, h}, {iT^o, • • • , 1^4} be the bases of Sym^l^, Sym W we defined in §2. 
Let {po,Pi,P2} be the dual basis of {^0,^1,^2}- Let f) be the Lie algebra of PGL(2). 
Then f) is the Lie algebra of SL(2) also. We consider F as a representation of f) 
also. Let A be the fundamental dominant weight of [). We denote the irreducible 
representation of f) with highest weight dA also by ciA. Then by considering weights, 
A^Vi ^ 6A ® 2A and 

(3.1) 6A (g) 2A ^ 8A © 6A © 4A, 2A 2A ^ 4A © 2A © k, 

where k is the trivial representation. 

Therefore, V contains the trivial representation precisely once. 

Note that V2 = f) as an f)-module. We identify Sym W* as the space of homo- 
geneous polynomials of degree four in two variables v = ''^ {vi V2) {v corresponds 
to i;iei -|- 1^262). We identify a = a{v) = aovf + • • • + a4t'2 with (uq, • • • , 04). 

Let 

(3.2) ^0 = ( Q J ) , i/i = ( J \).H2=l\ I 



An easy way to compute Lie algebra actions is to consider values in the ring of 
dual numbers A;[e]/(e^). 



Let 



(3.3) 



Ao 



Ai 



A, 



/O 1 0\ 

2 

3 

4 

Vo o/ 
^4000 

2 



0-2 

Vo 

/O 0\ 

4 

3 

2 

Vo 1 o/ 



A' 






-4/ 




A', 




A' 









Then the actions of f) on Fi, V2 are easy to describe and with respect to the bases 
{eo, • • • , 64}, {Iq, h, I2}, Ho, Hi, H2 map to 

iAo,A',),{Ai,A[),-{A2,A',) 

respectively. 

Since the action oi g E PGL(2) on a{v) G Sym W* is a{g~^v), the action of 
H E t) on a = a(v) is given by 

a((l - eH)v) = a{v) + e{Ha){v). 

Then easy computations show that 



(3.4) 



Hoa = (0, -4ao, -3ai, -2a2, -03) 
Hia = (— 4oo, — 2oi, 0, 2a3, ia^), 
H2a = {ai, 2a2, 3a3, 4o4, 0). 



For a = (ao, • • • , Q4), & = (^o, • • • , ^4), we define 
(3.5) 



Q{a, h) = aob4 - -aibs + -02^2 - -asbi + a4bo. 

4 6 4 



Then Q is a non-degenerate symmetric bilinear form, invariant under the action of 
PGL(2). This implies Q{Ha,b) is an alternating form for any H E i). We regard 
this alternating form as an element of A^(Sym^Vr*)* = A^Sym^VT. 

Lemma (3.6) The map H -^ fH{0',b) = Q{Ha,b) is an {)-homomorphism from \) 
to A^Sym^VT. 



Proof. Note that the action oi H E f) on an element f{a,b) in A^(Sym W*)* is 
given by {Hf){a, h) = -f{Ha, h) - f{a, Hh). So if H, H' e 1), 

{H'fH)ia,b) = -fH{H'a,b) - Ma^H'b) 
= -Q{HH'a, h) - Q{Ha, H'h) 
= -Q{HH'a, b) + Q{H'Ha, b) 
= Q{[H',H]a,b) 
= f[H',H]{a,b). 

D 

This defines an f)-homomorphism i) -^ A^Vi. Regarding this homomorphism as 
an element of A^Fi ® [)* = A^Vi ® V2 = F, we get a fixed point of V under the 
action of PGL(2). We compute this element explicitly. 

Note that the linear map defined by 

(3.7) Ho -^ -vl Hi -^ V1V2, H2 -^ -vf 

is an f)-homomorphism. 
By (3.4), 

(3.8) Q{Hoa, b) = a^b^ - -ai&2 + -02^1 - 03^0, 
Q{Hia, b) = -4aofe4 + -aibz - -03^1 + 4a4feo, 
Q{H2a, b) = 01^4 - -0263 + 2"3&2 - 04^1. 

Note that {ef , 2eie2, e|} is the dual basis of {i;^, i;ii;2, I'll' ^^^ (mo,t'f)4 = 
1, {mi^v\v2)A = |, etc. We identify A^Fi with the space of alternating bilinear 
forms on V{ by assuming 

m A m'(a, b) = (m, a)i{rn' , b)i — (to, b)i(m' , a)i 

for TO, to' e Fi, a,b E V{ . So by the correspondence H ^ fn, 

(3.9) iifo ^ 4too A TO3 - 12toi Ato2, 

Hi -^ — 4too a TO4 + 8toi A TO3, 
H2 -^ 4toi a TO4 — 12to2 a TO3. 

Since {7^0,-^1,-^2} corresponds to {^V2,viV2, ^vf} and {2/2,2/1,2/0} is its dual 
basis, this correspondence can be regarded as the element 8w where 

(3.10) w = (toq a 7712 — Srrii A TO2) <^ h + (—"^0 A TO4 + 2toi A TO3) h 

+ (mi A TO4 — 3to2 a ms) (g) /q. 

10 



These considerations show the following proposition. 
Proposition (3.11) The element w & V is fixed by PGL(2). 
In [19, p. 95], instead of w, the element 

w' = (mo A nil + 1712 A ms, mi A m2 + ma A m4, mo A m2 + mi A m.4) 

was considered (we shifted the indices in [19] because we are using indices 0, • • • ,4). 
However, by replacing mo,--- , m4 in w by m4, m2, mo, mi, ms respectively, and 
multiplying scalars to basis elements of A^Fi, we get the above element w' . There- 
fore, we are considering essentially the same element as in [19]. 

In [19, p. 96], the Lie algebra of G^, /T {G^, is the connected component 
of the stabilizer) is computed and is isomorphic to the Lie algebra of PGL(2). 
Therefore, this is the case for w also. Since we are assuming ch A; = 0, this implies 
G^ = PGL(2) X T if A; is algebraically closed (see [7]). For arbitrary A;, we still have 
the inclusion PGL(2) x T C G^. Since this is an isomorphism over A;, we get the 
following proposition 

Proposition (3.12) Gl, = PGL(2) x f . 

By the basis {eo, - - - , 64}, we regard V as the space of 5 x 5 alternating matrices 
with entries in V2. Then 

/ h -h\ 

-3^2 2Zi Iq 

w= Q U2 -3Zo 

-I2 -2li 3lo 

\ h -lo y 

By the definition (2.3), 

(3.13) Pfaffo(w;) = -3/g, Pfaffi(w;) = -3/o^i, Pfaff2(w;) = -2/? - loh, 
FisiSsiw) = -3/1/2, Pfaff4(ui) = -3ll 

Note that we are regarding them as elements of Sym^V2 and not Sym W. 

By the basis {Iq, hjh}, we regard 4>i{w) as a 3 x 3 matrix with entries in V^ as 
in 52. Then 



(3.14) (t>iH = - 

Let 

(3.15) no = /q, ni = ij, 712 = ij, n^ = /q/i, n^ = hh, n^ = loh G Sym^y2- 

Then {no, - - - , n^} is a basis of Sym V2. Let Hq, - - - , ns be elements of A^Sym V2 
Sym^F2* s^ich that n^ A n| = Sijno A- • ■ An^. Then 
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Smg 


\m\ 


\ml 


Iml 


2m\ 


\m% 


\ml 


\ml 


3m\ 



Pfaffo(«;) A FfsiSi{w) A Pfaff2(w;) A FisiSsiw) A PisiS^{w) 
= (-3no) A (-Sna) A (-2ni - ris) A (-3n4) A (-3n2) 
= — 3^no A ns A (2ni + ns) A n4 A n2 
= — 3^(2no A ris A rii A n4 A n2 + uq A ns A n^ A n4 A ^2) 
= 3^(nt -2n^). 

Therefore, 

(3.15) $i(w;) = nt-2n;. 

We identify Sym V2 with the dual space of Sym V2. Then with respect to the 
basis {p,pi,P2}, 'n'l,nl correspond top^,2poP2- Therefore, ^i{w) = p\ — ApQP2. We 
regard $i(w) as a 3 x 3 matrix as in §2. Then 



(3.16) ^i{w) 




7112 


-iK 


6mS 


2,ml 


-2m^ 


3mt 


6m| 


2 "^3 


m2 



Therefore, 



(3.17) 03 («;) = </.! («;)$!(«;) 



So, we get the following proposition easily. 

Proposition (3.18) Let a = aom,o + • • • + a^m,^. Then 

(1) ^2{w){a) = (tr((/)3(w;)2))(a) = 72(aoa4 - iOiQs + T^a|), 

(2) Fyj{a) = {det(f)3{w)){a) = 7200^204 + 9aia203 — 2a2 — 27aoal — 27ofa4. 

By these considerations, $1, $2 are non-trivial maps. By (3.16), the discriminant 
A(x) of ^i{x) is a non-zero polynomial. By Lemma (2.7), A(x) is a non-constant 
polynomial. Therefore, it is a relative invariant polynomial. So we reproved that 
w e F^^®. Since our case is known to be a regular prehomogeneous vector space, 
V^^ is a single Gfc^orbit if k is algebraically closed. Therefore, V^^ consists of x^s 
such that ^i{x) is non-degenerate. 

Since ^2{w) is non-degenerate, $2(2^) is non-degenerate for all x e F*^*^. Appar- 
ently, three components oiw are linearly independent. Therefore, $3(2;) G Gr(3, 10) 
is well defined for all x e V^^ also. 

§4 The orbit space Gk \ V^"" 

In this section, we prove that Gk \ V^^ corresponds bijectively with GL(l)fc x 
GL (3) A; -equivalence classes of ternary quadratic forms over k. 

We first recall the relation between the orbit space Gk \ V^^ and the Galois 
cohomology set. 
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For any algebraic group G over k, let H (k, G) be the first Galois cohomol- 
ogy set. We choose the definition so that trivial classes are those of the form 
{9~^9'^}aeGs\(k/k) (9 ^ ^k) ^^^ ^hc cocyclc condition is h^-j- = h-^h'^ for a continu- 
ous map {K}„^G^\{k/k) from Gal(A;/A;) to G^. 

Let {G,V) be an arbitrary regular prehomogeneous vector space, and w E V^^. 
Then for any x E V^^, there exists g^ E G^ such that x = QxW- Then c^ = 
{9x^9x}aeGaUk/k) determines a cohomology class in H {k, G^) and does not depend 
on the choice of Qx- The following theorem is due to Igusa [8]. 

Theorem (4.1) (Igusa) The correspondence 

Gk \ T^f 3x^CxE Ker(H^(A;, G^) -^ R\k, G)) 

is bijective. 

Note that Ker(H''^(A;, G^) -^ ii^{k, G)) is the set of elements c E ii.^{k, G^) which 
map to the trivial class in }i^{k,G). In our case, }i^{k,G) is trivial. Therefore, 
Gk\V^' = 'il\k,G^). 

We recall the correspondence between GL(l)fe x GL(3)fc-equivalence classes of 
ternary quadratic forms and quarternion algebras. Let V3 = Sym^V'2*- Then 
GL(V2) — GL(3) acts on V3 in the usual manner. We let GL(1) act on V3 by 
the usual multiplication. Then GL(1) x GL(3) acts on V3. Let {^05^15^2} and 
{po 5 Pi 5^2} be as before. Let w = p\ — 4poP2 5 and Q the corresponding quadratic 
form. It is well known (and is easy to verify) that the stabilizer of w is isomorphic 
to SO(Q) X GL(1) and SO(Q) ^ PGL(2). 

Therefore, (GL(l)fc x GL(3)fc) \F3f corresponds bijectively with H^(A;, PGL(2)). 
Since PGL(2) is isomorphic to the automorphism group of the associative algebra 
M(2,2), II''^(A;, PGL(2)) corresponds bijectively with isomorphism classes of quar- 
ternion algebras. Given a ternary quadratic form, the corresponding quarternion 
algebra is the Clifford algebra associated with the quadratic form. 

Now we go back to our situation. Let G, H^ V be as before. We consider the 
element w E V^^ which we defined in (3.10). We pointed out in (3.12) that G^ = 
PGL(2) X GL(1). 

Proposition (4.2) The group G^, is connected. 

Proof. We may assume that k is algebraically closed. Suppose g E Gk/T^. The 
connected component of the stabilizer of w in G/T is isomorphic to PGL(2). The 
conjugation by g induces an automorphism of PGL(2). Since there is no outer auto- 
morphism of PGL(2), by changing g if necessary, we may assume that g commutes 
with elements of PGL(2). Since Vi, V2 are irreducible representations, by Schur's 
lemma, g is represented by an element of the form (^1/5,^2-^3)- This element fixes 
w if and only if tlt2 = 1. So ^ = 1 (in G/f). D 

Proposition (4.3) (1) The map ^i : V ^ V^ = Sym^V^* induces a bijection 
G,\Ff -(GL(l),xGL(3),)\Kf|. 

(2) If X E V^^ , the projections of H^ to (ji,(j2 induce isomorphisms to the images. 
In particular, H^ — SO($i(a;)). 
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Proof. Let c G H {k,Gw). Then c becomes trivial in H {k,H) also. Let g = 
(91,92) G H]^ be the element such that c is represented by {9~^9'^}aeGa.i(k/k)- Then 
the orbit in V^^ corresponding to c is gw. By Lemma (2.7), ^i{gw) = 92^i{w). 

Since H^(A;,G^) ^ H^(A;, PGL(2)) and the projection of PGL(2) to G2 is an 
isomorphism to its image, 

ii\k,G^) 3 {^"Vl.GGaKfcA) ^ {^2"'^2}.GGal(fcA) ^ H^(fc,PGL(2)) 

is a bijection. Note that we are considering PGL(2) C G for the first element 
and PGL(2) C G2 for the second element. Since (GL(l)fc x GL(3)fc) \ y|| ^ 
H^(A;,PGL(2)), this proves (1). 

Note that (PGL(2) xf)nH = PGL(2). So H^ ^ PGL(2). We already pointed 
out that statement (2) holds for w in §2. Let x G V^^. By Lemma (2.7), the 
projection of H^ to G2 is contained in SO($i(a;)). So it is enough to prove (2) 
when k is algebraically closed. But then x is in the orbit of w and (2) follows 
easily. D 

Remark (4.4) The map $1 induces a map Gk \ V^^ -^ GL(3)a; \ V^^ also, but this 
may not be surjective. This may be regarded as the section (GL(l);fc x GL(3)fc) \ 
Vi^ -^ GL(3)fc \ y|| defined by x ^ (deta;)-^^. 

§5 Intermediate groups 

Let X e V^^. By Proposition (4.3), H^r is connected in classical topology. So 
H^R-\- = -^xK- If ^1(3^) is definite, H^r is compact by Proposition (4.3) also. Then 
HxrHi C Hr/Hz is a compact set. Therefore, an analogue of the Oppenheim 
conjecture is not applicable to such points. The set of real indefinite non-degenerate 
ternary quadratic forms is a single GL(1)k x GL(3)K-orbit. Therefore, we only 
consider x € GrW. 

We determine all the closed connected subgroups between H^r_^_ and Hr for 
all X e Grw for the rest of this section. This reduces to the consideration of 
Lie algebras. We consider an arbitrary ground field k of characteristic zero and 
specialize to /;; = M in (5.10). 

We first describe possible candidates for such subgroups. By Lemmas (2.7), 
(2.13), (2.16), $i,$2,^3 are iif-equivariant maps. As we pointed out at the end 
of §3, ^i{x) e Sym V^ , ^2{x) G Sym V2 are non-degenerate for x G GkW. So let 
SO($i(a;)), S0($2(a^)) be the corresponding special orthogonal groups. 

In the following definition, x € GkW. 

Definition (5.1) (1) H^i C GL(Vi),i7a;2 C GL(V2) are the images of the projec- 
tions of Hx to Gi,(j2 respectively. 

(2)i/,3 = SO($2(a^))cGL(Fi). 

Note that both Hxi,Hx2 are isomorphic to H^, and Hx = PGL(2). 

Let f) be the Lie algebra of PGL(2) as before. Let f)i = sl(5),f)2 = sl(3) (Lie 
algebras of SL(5), SL(3)). If f is a Lie algebra between f) and f)i x f)2, it it is an 
f)-module. So we first decompose f)i, f)2 to direct sums of irreducible {)-modules. 
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Let 



(5.2) 



B = Bibo, 



C = Cico, 



C6 



D = D{do,--- ,d8) 



( 2^2 


-3bi 


&o 








\ 


12fe3 


-b2 


-2fei 


3feo 





664 


3^3 


-2^2 


3fei 


660 





3^4 


-2^3 


-b2 


12bi 


V 





b4 


-3&3 


2b2 J 


/ C3 


3C2 


-ci 


Co 


\ 


12C4 


-2C3 


— 4C2 





Acq 


6C5 


— 6C4 





-6C2 


6ci 


4C6 





-4C4 


2C3 


12C2 


V 


C6 


-C5 


3C4 


-cj 


/ d^ 


-ds 


fi2 


-dl 


do \ 




4^5 


-4^4 


4c?3 


-4d2 


4di 




6^6 


-64 


6^4 


-Qds 


6^2 


1 


4^7 


-4^6 


M5 


-M4 


4t^3 




V d^ 


-dr 


de 


-4 


d4 / 





fe'o 



5'(feo, 



-b[ fe'o 



2fe'3 -2fe'2 2fe; 



b'. 



-fe', ft'o 



where feo • ■ • G ^■• 










We define 










(5.3) 


U2 


= {Bibo,-- 


■ ,h) \ bo,- ■ 


• ,64 G k}, 




Us 


= {C{co,-- 


■ ,cq) Co, • • 


• ,C6 e k}, 




U4 


= {D{do, ■ 


• ,C^8) C^O,- 


■■ ,d8ek} 




V2-- 


= {B'{b'„- 


• ^^4) 1 ^or- 


■ ,b'^ek}. 



Let Ui, Vi be the images of {) in [)i, [)2- Ui, Vi are clearly, sub f)-modules. 

Lemma (5.4) The subspaces U2, Us, t/4, V2 are irreducible sub \)-modules with high- 
est weights 4A, 6A, 8A, 4A respectively. 



Proof. By straightforward computations. 



(5.5) 


[Ao,5(l,0,---,0)] = 


= [Ao,C(l,0,---,0)] = [Ao,^(l,0,---, 




'A'^,B' {!,{), ■■■,{))] 


= 0, 




[Ai,5(l,0,---,0)] = 


= 45(l,0,---,0). 




[Ai,C(l,0,---,0)] = 


= 6C(1,0, ••• ,0), 




:Ai,i)(l,0,---,0)] = 


= 8£>(1,0, ••• ,0), 




[^;,5'(l,0,---,0)] 


= 45'(l,0,---,0). 


Alsc 


), 




(5.6) 


[A2,5(6o,--- 


,b^)] = B{Q,bo,&bi,b2,Abs), 




[A2,C(C0,--- 


, ce)] = C(0, 2co, ci, -12c2, C3, IOC4, 2>c^) 




[A2,D{do,--- 


,ds)]=D{0,do,2di,3d2,--- ,8d7). 




[A'2,B'{b'„... 


,b'^)] = B'iO,b'o,2b[,3b'„4b',). 



0)] = 0, 
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The author used MAPLE [1] to find 1/2,113,1/4 but computed (5.5), (5.6) man- 
ually and checked (5.5), (5.6) by MAPLE again. 

By (5.6), 1/2 is spanned by elements of the form ad(A2)'5(l, 0, 0, 0, 0) (ad(*) is 
the adjoint representation). Since 



ad{Ao)B{l, 0, 0, 0, 0) = 0, ad{Ai)B{l, 0, 0, 0, 0) = 45(1, 0, 0, 0, 0), 

1/2 is an irreducible sub f)-module with highest weight 4A. 
Other cases are similar. D 

Proposition (5.7) (1) [U2,U2] = Ui® U3. 

(2) [U2,U3] = U2(BU4. 

(3) [U2,U4] = U3. 

(4) [U3,U3] = Ui(BU3. 

(5) [U3,U4\ = U2(BU4. 

(6) [U4,U4] = Ui®Us. 

(7) [F2,T^2] = Fi. 

Proof. We first consider (1). Since U2 is irreducible, for any non-zero element 

X G 1/2, 1/2 is generated by X as an P)-module. So [U2, U2] is generated by [X, U2] 
as an f)-module also. 

By straightforward computations, 
(5.8) [5(0,0,l,0,0),5(feo,---,M] 

+ C(0,-4feo,-^, 0,-^,-4^4,0). 

We chose 5(0, 0, 1, 0, 0) because it is diagonal. 

By (5.8), [U2, U2] cUi® U3. By choosing fei = ^3 = in (5.8), [U2, U2] contains 
a non-zero element of U3. This implies [U2, U2] contains U3. By choosing fei 7^ in 
(5.8), [1/2, 1/2] contains an element of the form X + X' where X E Ui is non-zero 
and X' e 1/3. So X G [1/2,1/2]- This implies [1/2,1/2] contains Ui also. This proves 
(!)• 

Other cases follow from the following relations and by similar arguments. We 
found these relations manually. However, it can be checked by a routine program 
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in MAPLE (which we did). 

[5(0,0,l,0,0),C(co,---,C6)] 

16ci 16c2 ^ 16c4 I6C5 
= ^{ ^> ^' ' ^' ^^ 

I n/n Q 12ci 15c2 15c4 12c5 
+ i5(0, -3co, ^, ^, 0, —, —, -3c6, 0), 

[B{0,0,l,0,0),D{do,--- ,ds)] 

= C(— 3c?i, —4^2, —(^3, 0, — (i5, —4(^6, —3^7), 

[C(0,0,0,l,0,0,0),C(co,---,C6)], 

= 6C2A0 - QC4A2 + C(-co, ci, C2, 0, -C4, -C5, ce), 

[C;(0,0,0, l,0,0,0),D(fio,--- ,c?8)] 

= B[—j^,^^,^, — ^, — ^j 

+ 15(24, -c?i, —, — ^, 0, — , -^, dr, -2d8), 

[L»(0,0,0,0,l,0,0,0,0),i5(do,---,(^8)] 

= —IMsAq — 14(^5^2 + C{—5di, 5c?2, Sds, 0, 3c?5, 5c?6, — 5^7), 

[5'(0,0,l,0,0),i?'(6[„ •••,&!)] 

= -3fei.4'o - 3fe3^2- 

D 

Note that the Lie algebras of Hwi,Hw2 are isomorphic to sl(2). We denote 
the Lie algebra of Hy,3 by so(5) (more precisely so(3,2)). Since dimso(5) = 10, 
so(5) = t/i ® t/3 by counting the dimension. 

Proposition (5.9) // f) C f C f)i x f)2 is a Lie subalgebra, f) is one of the following 
suhalgebras. 

\), sl(2) X sl(2), sl(2) X sl(3), so(5) x sl(2), 

so(5) X sl(3), sl(5) X sl(2), sl(5) x sl(3). 

Proof. Let f be as above. Note that sl(3) does not contain any f)-module which is 
isomorphic to t/3 or [/4. Suppose f D t/4. Then f D t/i ® f/3 by Lemma (5.7) (1). So 
f D U2 by Lemma (5.7) (5). Since f D f/i, f D f/i ® ^i. Therefore, f D sl(5) x sl(2). 
So f = sl(5) X sl(2) or sl(5) x sl(3). 

Suppose the projection of f to the first factor contains U2- Then there exists 
an f)-homomorphism a : U2 ^ V2 such that {x^a{x)) G f for all x G U2- By 
Lemma (5.7) (1), the projection of f to the first factor contains t/3. Since t/3 is not 
equivalent to any other factor, f contains t/3. By Lemma (5.7) (4), f D t/i. Since 
f D [), f D t/i®yi. If X e t/2,2/ e t/i, (?/,0) e f. So [(2/,0), {x,a{x))] = ([2/,x],0) G f. 
Since [^^1,^/2] = ^^2, f ^ C^2- By Lemma (5.7)(2), f D t/4 and it reduces to the 
previous case. 

Suppose f does not contain t/4 and the projection to the first factor does not 
contain U2- Suppose f D Uz- By Lemma (5.7)(4), f D Ui. Therefore, f has so(5) as 
the first factor. This implies f = so(5) x sl(2) or so(5) x sl(3). 
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Suppose the projection of f to the first factor is Ui. If f D V25 f ^ ^i also. 
Therefore, f = sl(2) x sl(3). Otherwise the projection of f to both factors are sl(2). 
Since there is no sub f)-module between f) and t/i x Vi, f is f) or sl(2) x sl(2). D 

Now we specialize to the field k = M.. 

Proposition (5.10) Let x e G^w and Hs;^. C F C H^. be a closed connected 
subgroup. Then F is one of the following subgroups. 

HxR, HxiR X Hx2R, HxiR X SL(3)k, 

Hx3R X Hx2R, Hx3R X SL(3)r, SL(5)r X Hx2R, SL(5)k X SL(3)k. 

Proof. If X = gw for g G Gr, H^r = gH^^g~^, etc. So we may assume that x = w. 
Then this proposition follows from the previous proposition. D 

§6 An analogue of the Oppenheim conjecture 

In this section, we prove an analogue of the Oppenheim conjecture. 

In the following lemma, x G Vff^. We define H^ic, etc. as in Definition (5.1). 

Lemma (6.1) (1) If y E Vc is fixed by H^c, y is a scalar multiple of x. 

(2) 7/2/ e Gr(3, 10)c is fixed by H^ic, y = ^3{x). 

(3) If y E Sym^V'2* i^ fixed by Hx2C, y is a scalar multiple of ^i{x). 

(4) If y E Sym V{ is fixed by H^zc, y is a scalar multiple of ^2{x). 

Proof. Consider (1). Let x = gw with g G Gc- Then H^c = 9Hwc9~^, and g~^y 
is fixed by H^c. So we may assume x = w. By (3.1), V contains the trivial 
representation of f) precisely once. Therefore, the set of fixed points of H^£ is of 
dimension one. This proves (1). 

Next consider (2). Let x = gw with g = {gi,g2) £ Gc again. Let S C A^Fic 
be the three dimensional subspace corresponding to y. Then g~^S = gi^S is fixed 
by gi^H^icdi = H^ic- Note that G2 acts trivially on Gr(3, 10). So gi^S is H^c- 
invariant also. Therefore, gi^S is a sub f)c^niodule. However, A^Vic = 6A ® 2A. 
So there is only one three dimensional sub fic^niodule. Since S^ also satisfies this 
condition, y = gi[Sj\ = <^z{x). 

Statements (3) , (4) are well known and were used in the proof of the Oppenheim 
conjecture for quadratic forms. D 

In the following theorem, let x G Gk.w. Then i7^R_|_ is generated by unipotent 
elements. Let Hxr C F C H-r be the closed connected subgroup such that H^rHz = 
FHi. By Ratner's theorem, such F exists. 

Theorem (6.2) (1) If^2{x) i P(SymVi*)Q and^^ix) ^ Gr(3, 10)q, F = SL(5)rX 
Hx2R or F = SL(5)k x SL(3)k. 

(2) //$i(x) ^ P(SymV2*)Q, $2(2;) ^ P(SymVi*)Q, and ^s{x) ^ Gr(3, 10)q, F = 
SL(5)r X SL(3)r. 

Proof. Suppose F = H^r- Then F is defined over Q. Therefore, for any a G 
Aut(C/Q), H^^ = Hxc- Since H^^ = Hx<^c, x'^ is fixed by H^c- So x'^ is a scalar 
multiple of x by Lemma (6.1). Since this is the case for all a, [x] G P(y)Q. Since 
$1, $2, ^3 are defined over Q, ^i{x), $2(2^), ^six) are all Q-rational points. 
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Suppose F = Hj,iBi X H^2R, or i/^iK x SL(3)r. We show that ^six) G Gr(3, 10)$ 
Since the argument is similar, we only consider the first case. 
For any a E Aut(C/Q), 



^ailC ^ -^a;2C 



H. 



ai-^lC 



xH. 



a;'^2C 



HxlC X Hx2C- 



Since G2 acts trivially on Gr(3, 10), $3(3;'^) is fixed by Hxic- This implies $3(3;'^) = 
$3(3;). Since $3 is defined over Q, ^^{x'^) = ^^{xY = $3(2;) by Lemma (6.1). 
Therefore, $3(3;) e Gr(3, 10)q. 

By a similar argument, if F = H^m x H^2R,H3r x H^2R, or SL(5)k x H^2R, 
[$i(a:)] e P(SymV2*)Q. Also if F = H^sr x H,2R or iy,3R x SL(3)m, [M^]] e 
P(SymVi*)Q. 

By these considerations, conditions in (1), (2) force F to become the given 
subgroups. D 

Lemma (6.3) Let x e Gk.w. Then for any non-zero real number r, there exists 
h e H-R and a primitive integer point a G Viz such that Fy^-i^{a) = r. 



Proof. We may assume x = Xw where A e M \ {0}. Since F\h-i^{a) = X^^Fy,{ha) 

■t-^ 0' 



the above condition is equivalent to Fyj{ha) = A ^^r. Put t = —A ^^{^)^ ■ Then 



h 



/ /t~' 0\ \ 

10 

t 

10 

VV l/ J 



and a = *(0 1 0) satisfy the condition. D 

In the following theorem, x E Grw. 

Theorem (6.4) // [$2(2^)] ^ P(SymVi*)Q, and $3(2^) ^ Gr(3, 10)q, the set of 
values of the cubic form Fx{a) at primitive integer points is dense in M. 

Proof. Let r be a non-zero real number. We choose h = {h' , h") e H-^ and a e Viz 
as in Lemma (6.3). By Theorem (6.2), there exist hi = {h'l^h'l) G Hx^ and /i2 = 
(/i2, /i2) ^ Hi such that h'-^h'2 is close to h' . Then 

Fx{h'2a) = F^^i^ia) = F^^i^-i^{a) = F(^^-i/,i,i)^(a) 



is close to 



^(/i'-i,l)a)(a) — ^/i-ia)(a) 



Note that Fy^-i^ does not depend on the second component of h. 
Since h'2a G Vii is primitive, this proves the theorem. D 

Note that if a; = gw with g = {gi,g2) G Gr, 

^3{x) = gi^:,{w), [^2{x)]=gi[^2{w)l F^{a) = {deigi)^''{deig2fF^{g-^a). 
Therefore, writing down F„, etc. explicitly, we get the statement of Theorem (0.2). 
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